A Hybrid Approach for Entity Extraction in Code-Mixed Social Media Data

Entity extraction is one of the important tasks in various natural language processing (NLP) application areas. There has been a significant amount of works related to entity extraction, but mostly for a few languages (such as English, some European languages and few Asian languages) and doamins such as newswire. Nowadays social media have become a convenient and powerful way to express one’s opinion and sentiment. India is a diverse country with a lot of linguistic and cultural variations. Texts written in social media are informal in nature, and perople often use more than one script while writing. User generated content such as tweets, blogs and personal websites of people are written using Roman script or sometimes users may use both Roman as well as indigenous scripts. Entity extraction is, in general, a more challenging task for such an informal text, and mixing of codes further complicates the process. In this paper, we propose a hybrid approah for enity extraction from code mixed language pairs such as English-Hindi and EnglishTamil. We use a rich linguistic feature set to train Conditional Random Field (CRF) classifier. The output of classifier is post-processed with a carefully hand-crafted feature set. The proposed system achieve the F-scores of 62.17% and 44.12% for English-Hindi and English-Tamil language pairs, respectively. Our system attains the best F-score among all the systems submitted in Fire 2016 shared task for the English-Tamil language pairs. CCS Concepts •Computing methodologies→Natural Language Processing; •Information System → Information Extraction; •Algorithm → Conditional Random Field(CRF);